Wearable pupil-forming display apparatus

ABSTRACT

A wearable display apparatus is described herein. The wearable display apparatus includes an image generator that forms a 2D image, a partially transmissive mirror having a curved reflective surface, a beam splitter disposed to reflect light toward the curved mirror surface, and an optical image relay that is configured to conjugate the formed 2D image at the image generator to a curved focal surface of the partially transmissive mirror.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/496,102, filed Oct. 7, 2021, which is a continuation of U.S. patentapplication Ser. No. 17/389,484, filed Jul. 30, 2021 (now U.S. Pat. No.11,181,747, issued Nov. 23, 2021), which is a continuation of U.S.patent application Ser. No. 17/139,167, filed Dec. 31, 2020 (now U.S.Pat. No. 11,112,611, issued Sep. 7, 2021), which claims the benefit ofU.S. Provisional application Ser. No. 63/060,343, filed on Aug. 3, 2020,the disclosures of which are hereby incorporated by reference in theirentirety and for all purposes.

TECHNICAL FIELD

The present disclosure generally relates to wearable display apparatusand more particularly to a wearable display device that providesaugmented reality (AR) viewing with a catadioptric pupil-forming opticalsystem that renders a binocular 3D virtual image from a pair of2-dimensional (2D) displays.

BACKGROUND

Virtual image display has advantages for augmented reality (AR)presentation, including providing the capability for display of imagecontent using a compact optical system that can be mounted on eyeglassesor goggles, generally positioned very close to the eye (Near-EyeDisplay) and allowing see-through vision, not obstructing the view ofthe outside world. Among virtual image display solutions for AR viewingare catadioptric optics that employ a partially transmissive curvedmirror for directing image-bearing light to the viewer's eye and apartially reflective beam splitter for combining light generated at a 2Ddisplay with the real-world visible scene which forms a 3D image whenviewed binocularly.

Vision correction applications have employed wearable display devices inorder to enhance or compensate for loss of vision over portions of asubject's field of view (FOV). Support for these types of applicationscan require additional components and can introduce various factorsrelated to wearability and usability that contribute to the overallcomplexity of the optical design and packaging.

Among challenges that must be addressed with wearable AR devices isobtaining sufficient brightness of the virtual image. Many types of ARsystems, particularly those using pupil expansion, have reducedbrightness and power efficiency. Measured in NITS or candelas per squaremeter (Cd/m2), brightness for the augmented imaging channel must besufficient for visibility under some demanding conditions, such asvisible when overlaid against a bright outdoor scene. Other opticalshortcomings of typical AR display solutions include distortion, reducedsee-through transmission, small eye box, and angular field of view (FOV)constraints.

Some types of AR solution employ pupil expansion as a technique forenlarging the viewer eye-box. However, pupil expansion techniques tendto overfill the viewer pupil which wastes light, providing reducedbrightness, compromised resolution, and lower overall image quality.

Challenging physical and dimensional constraints with wearable ARapparatus include limits on component size and positioning and, withmany types of optical systems, the practical requirement for folding theoptical path in order that the imaging system components beergonomically disposed, unobtrusive, and aesthetically acceptable inappearance. Among aesthetic aspects, compactness is desirable, withlarger horizontal than vertical dimensions.

Other practical considerations relate to positioning of the displaycomponents themselves. Organic Light-Emitting Diode (OLED) displays havea number of advantages for brightness and overall image quality, but cangenerate perceptible amounts of heat. For this reason, it is advisableto provide some distance and air space between an OLED display and theskin, particularly since it may be necessary to position these devicesnear the viewer's temples.

Still other considerations relate to differences between users of thewearable display, such as with respect to inter-pupil distance (IPD) andother variables related to the viewer's vision. Further, problemsrelated to conflict between vergence depth and accommodation have notbeen adequately understood or addressed in the art.

It has proved challenging to wearable display designers to provide theneeded image quality, while at the same time allowing the wearabledisplay device to be comfortable and aesthetically pleasing and to allowmaximum see-through visibility. In addition, the design of system opticsmust allow wearer comfort in social situations, without awkwardappearance that might discourage use in public. Providing suitablecomponent housing for wearable eyeglass display devices has proved to bea challenge, making some compromises necessary. As noted previously, inorder to meet ergonomic and other practical requirements, some foldingof the optical path along one or both vertical and horizontal axes maybe desirable.

SUMMARY OF INVENTION

The Applicants address the problem of advancing the art of AR displayand addressing shortcomings of other proposed solutions, as outlinedpreviously in the background section.

The Applicants' solution uses pupil forming and can be distinguishedfrom pupil expansion systems known to those skilled in the art. Bycomparison with pupil expansion approaches, the Applicants' approachyields a more efficient optical system with improved image quality.Moreover, the eyes of the viewer can clearly see and be seen by others,with minimal impediment from the optics that provide the electronicallygenerated virtual image.

Wearable display apparatus of the present disclosure are well-adaptedfor systems that complement viewer capabilities, such as where a viewermay have visual constraints due to macular degeneration or othercondition of the eyes.

With these objects in mind, there is provided a wearable displayapparatus comprising a wearable display apparatus comprising a headsetthat is configured for display from a left-eye optical system and aright-eye optical system, wherein each optical system defines acorresponding exit pupil for a viewer along a view axis and comprises:(a) an electroluminescent image generator that is energizable to directimage bearing light for a 2D image from an emissive surface; (b) acurved reflective surface disposed along the view axis and partiallytransmissive, wherein the curved reflective surface defines a curvedintermediate focal surface; (c) a beam splitter disposed along the viewaxis and oriented to reflect light toward the curved reflective surface;(d) an optical image relay that is configured to optically conjugate theformed 2D image at the image generator with the intermediate focalsurface, wherein the optical image relay comprises: (i) a prism havingan input surface facing toward the emissive surface of the imagegenerator, an output surface facing toward the intermediate focal plane,and a folding surface extending between the input and output surfacesand configured for folding an optical path for light generated by theimage generator, wherein an aperture stop for the relay lies within theprism; (ii) at least a first plano-aspheric lens in optical contactagainst the prism input surface and configured to guide theimage-bearing light from the image generator toward the folding surface;wherein the relay, curved mirror, and beam splitter are configured toform the exit pupil for viewing the generated 2D image superimposed on aportion of a visible object scene, wherein combined images from bothleft- and right-eye optical systems form a 3D image for the viewer; and(e) a plurality of sensors coupled to the headset and configured toacquire measured data relating to the viewer.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic front view showing placement of opticalcomponents of the system.

FIG. 1B is a schematic side view showing placement of optical componentsof the system.

FIG. 2A is a schematic front view showing the optical path throughcomponents of the system.

FIG. 2B is a schematic side view showing the optical path throughcomponents of the system.

FIG. 3A is a schematic that shows, in perspective view, components of anoptical apparatus for AR viewing.

FIG. 3B is a simplified schematic of FIG. 3A.

FIG. 4 is a schematic that shows, from an alternate perspective view,components of an optical apparatus for AR viewing.

FIG. 5 is a side view schematic of an image relay.

FIGS. 6A, 6B are schematic, showing the image relay and components forforming the exit pupil.

FIG. 6C is a schematic view that shows an alternate embodiment of thepresent disclosure that employs the imaging path for eye tracking.

FIGS. 7A-7C show various features of an embodiment useful forcompensating for macular degeneration.

FIG. 8 is a block diagram illustration showing integrated components foran HMD system in an exemplary embodiment.

FIG. 9 is a perspective view that shows positions of various sensors,processor, and components of the head-mounted display according to anembodiment.

FIG. 10 is a schematic diagram that shows components of an interpupildistance adjustment system for the head-mounted display.

FIG. 11A is a schematic diagram showing optical relay components usedfor focal plane adjustment.

FIG. 11B is a perspective view that shows position of an actuatorrelative to the display and to corresponding optics.

FIG. 11C is a perspective view from behind the display component,showing a piezoelectric actuator mounted to a plate behind the display.

FIG. 11D is a side view showing components of a dynamic focus adjustmentapparatus.

FIG. 12 is a schematic diagram showing visual accommodation andvergence-accommodation conflict.

FIG. 13 is a schematic diagram that shows aspects of dithering geometryfor enhanced image resolution.

DETAILED DESCRIPTION

The following is a detailed description of the preferred embodiments ofthe disclosure, reference being made to the drawings in which the samereference numerals identify the same elements of structure in each ofthe several figures.

While the devices and methods have been described with a certain degreeof particularity, it is to be noted that many modifications may be madein the details of the construction and the arrangement of the devicesand components without departing from the spirit and scope of thisdisclosure. It is understood that the devices and methods are notlimited to the embodiments set forth herein for purposes ofexemplification. It will be apparent to one having ordinary skill in theart that the specific detail need not be employed to practice accordingto the present disclosure. In other instances, well-known materials ormethods have not been described in detail in order to avoid obscuringthe present disclosure.

Reference throughout this specification to “one embodiment,” “anembodiment,” “one example,” or “an example” means that a particularfeature, structure, or characteristic described in connection with theembodiment or example is included in at least one embodiment of thepresent disclosure. Thus, appearances of the phrases “in oneembodiment,” “in an embodiment,” “one example,” or “an example” invarious places throughout this specification are not necessarily allreferring to the same embodiment or example. Furthermore, the particularfeatures, structures, or characteristics may be combined in any suitablecombinations and/or sub-combinations in one or more embodiments orexamples.

In the context of the present disclosure, the term “eyebox” has itsconventional meaning in the HMD arts, as functionally equivalent to “eyemotion box” and similar phrases. The eyebox is that volume of spacewithin which the viewable image is formed by an optical system or visualdisplay. When the viewer's pupil is within this volume, the viewer cansee all of the generated display content; with the pupil is outside ofthis volume, the user is typically not able to view at least someportion of the display content. A larger eyebox is generally desirable,as this allows for lateral and axial movement of the eye, while stillmaintaining a full field of view. The size of the eyebox relatesdirectly to the size of the exit pupil for a display system.

Several (or different) elements discussed herein and/or claimed aredescribed as being “coupled,” “in communication with,” “integrated,” or“configured to be in communication with” or a “system” or “subsystem”thereof. This terminology is intended to be non-limiting and, whereappropriate, be interpreted to include, without limitation, wired andwireless communication using any one or a plurality of a suitableprotocols, as well as communication methods that are constantlymaintained, are made on a periodic basis, and/or made or initiated on anas-needed basis.

Where they are used, the terms “first”, “second”, and so on, do notnecessarily denote any ordinal, sequential, or priority relation, butare simply used to more clearly distinguish one element or set ofelements from another, unless specified otherwise.

In the context of the present disclosure, the term “coupled” is intendedto indicate a mechanical association, connection, relation, or linking,between two or more components, such that the disposition of onecomponent affects the spatial disposition of a component to which it iscoupled. For mechanical coupling, two components need not be in directcontact, but can be linked through one or more intermediary components.

An embodiment of the present disclosure provides AR viewing and displayhaving a large FOV with an optical system having an optical path thatfolds in the horizontal or x-direction, the direction substantiallyparallel (+/−15 degrees) to a line between left and right pupils of aviewer, for forming an intermediate image to the curved mirror. Anembodiment of the AR system of the present disclosure has a componentarrangement as shown schematically in the front view of FIG. 1A and fromthe side in FIG. 1B. The corresponding light path is shown schematicallyin FIGS. 2A and 2B, respectively. A flat-panel display is energized asan image generator 10 to form an image and to direct image-bearing lightthrough beam-shaping optics and to a folding prism 20 that redirects theimage-bearing light towards a beam splitter 24 and to a curved mirror 30for forming the virtual image from electronically generated imagecontent. Image generator 10 can be a display that emits light, such asan organic light-emitting device (OLED) array or a liquid crystal arrayor a micro-LED array with accompanying lenslets, or some other type ofspatial light modulator useful for image generation.

In order to address the need for improved overall imaging performance,wider FOV, compactness, and other factors outlined in the background,embodiments of the present disclosure have a number of features shownparticularly in FIGS. 3A, 3B, and 4 . Specific features of interestinclude: (i) relay of the image generator 10 to form a curvedintermediate image I as a conjugate image. As a type of “aerial” image,intermediate image I is formed in air, serving as the optical “object”for forming the virtual image. Intermediate image I is formed along thecurved focal surface of curved mirror 30, with the approximate positionshown by a dashed line in FIG. 5 . An optical relay 40, with particularstructure as described in more detail subsequently, conjugates the imageformed from image generator 10 to the curved intermediate image I alongthe focal surface. Curved mirror 30 is partially transmissive, such asbetween about 30% to 70% transmissive, for example, allowing visibilityof the real-world object scene to the viewer. A nominal transmissionrange of about 50% is useful in many applications. (ii) use of acylindrically curved quarter-wave plate (QWP) between mirror 30 and beamsplitter 24. Curvature of this element helps to reduce variations of theretardation imparted to the image-bearing light by the QWP over thefield of view. (iii) large exit pupil 44. System optics can form a 10 mmexit pupil at the viewer's eye-box for eye E. Forming a suitably sizedpupil for the viewer helps to provide an eyebox of reasonable dimensionsto allow eye movement, without noticeable vignetting. Also, an enlargedeyebox permits the headset to move or slip without noticeabledegradation of the viewed image(s). The apparatus does not need toprovide pupil expansion, such as is used in existing wearable displayapparatus, but uses pupil-forming optics for improved efficiency andbrightness, as well as for improved image resolution.

Significantly, the eyes of the viewer can clearly see and be seen byothers, with minimal impediment from the beam splitter and curved mirroroptics that provide the electronically generated virtual image.

With the optical arrangement shown, the aperture stop AS lies withinprism 20 of the image relay, along or very near the fold surface that isprovided. This arrangement is advantageous for component packaging andspacing, allowing prism 20 to be reduced in size over otherconfigurations using a folding prism.

The given design allows an FOV along the horizontal (x) axis, the axisparallel to a line between left and right pupils of the viewer's eyes,of greater than 50 degrees. The FOV aspect ratio (horizontal:vertical)equals or exceeds 1.5. Digital correction is not needed for distortionor lateral color.

According to an embodiment, curved mirror 30 has a conic surface shape.The conic shape is advantaged, in the embodiment shown herein, helpingto control chief ray angles, thus correcting for distortion.

Depending on whether or not polarization is used for configuring lightpaths, beam splitter 24 can be either a polarization-neutral beamsplitter or a polarization beam splitter. Beam splitter 24 can be, forexample, a wire grid polarization beam splitter as shown in FIG. 4 .

Image Relay 40

FIG. 5 shows an enlarged side view of relay 40 for relay of the displayimage from an electroluminescent display (such as an OLED in theexamples shown) to the focal surface position of mirror 30 (shown in aperspective view in FIG. 4 ) and for shaping the relayed image tosuitable curvature to correct distortion. A concave-plano field lens LI,with sides truncated along the vertical direction as shown in FIG. 4 inorder to reduce weight and provide a more compact system, shapes thelight from OLED display image generator 10, providing a beam to ameniscus singlet lens L2.

From lens L2, the imaging light goes to a doublet L3 having aconcave/convex flint glass lens cemented to a crown glass lens.

An aspheric plano-convex lens L4 is in optical contact with the inputface of prism 20, such as cemented to prism 20. A second plano-asphericlens L5 can be cemented to the output face of prism 20. This cementedarrangement facilitates alignment of these optical components. Accordingto an alternate embodiment, only a single plano-aspheric lens L4 isdeployed at the prism 20 input surface.

The turning surface 26 of prism 20 is coated to enhance reflection.Hypotenuse or turning surface 26 of the prism is essentially the relay(and system) aperture stop.

Intermediate image I is formed in the shape and location of the focalsurface of the curved mirror. Proceeding backward along the optical pathfrom intermediate image I are the following components: Plano-aspherelens L5; Folding prism 20 with turning surface 26; plano-asphere lensL4; Doublet L3; Meniscus singlet L2; Field lens LI; Image source orgenerator, display 10

FIG. 6A and simplified FIG. 6B show an alternate view of the displayoptics from exit pupil 44. Chief rays are shown in FIG. 6B; these chiefrays converge at the position of exit pupil 44 at eye E. FIG. 6B alsoshows the approximate position of an intermediate pupil P_(i) at theaperture stop, near the folding surface 26 of prism 20.

As shown in FIGS. 1A and 4 , the image generator is disposed to directimage-bearing light beam in a horizontal direction and along a path thatlies above eye-level, as the display optics are normally worn by asitting or standing viewer. Prism 20 can be tilted slightly away fromthe forehead of the viewer, to direct light in front of the face at anoblique angle to vertical, as shown in the embodiment of FIG. 2B.

The layout and routing of the optical path are particularly suitable forproviding augmented reality 2D and 3D viewing in a wearable device.Using relay 40 allows the positioning of image generator 10 to be out ofthe direct field of view; in addition, relay 40 allows image generator10 to be positioned at sufficient distance away from the skin surface toavoid contact and consequent discomfort. The use of a first x-direction(horizontal) fold, followed by a y-direction (vertical) folding enablesthe imaging optics to be compactly packaged with reasonable optical pathdistance to allow a measure of light beam shaping and correction. Prism20 can be rotated over at least a range of angles about the x axis,allowing a measure of alignment as well as adaptation to differentmirror 30 curvatures. Employing a curved surface for an optional QWPcomponent helps to reduce variations, over the FOV, of retardationimparted by the QWP; excessive variation over the field may otherwisecause some brightness fall-off.

Using a wire-grid polarizer reduces light loss, allowing high levels ofvisibility to the external, real-world object scene content, along withreduced light leakage over other polarization components.

Image source 10 may be unpolarized. In one embodiment, a polarizing beamsplitter is used, such as a wire grid splitter made by Moxtek, Inc.,Orem, Utah. This type of beam splitter reflects only one polarization,usually S polarization, towards the conic combiner. The orthogonalpolarization, P polarization, is transmitted and is absorbed (absorbernot shown). To prevent the small amount of P light from being reflected,an optional polarizer can be placed at the image source.

The mirror 30 provides a conic combiner in embodiments shown, with poweronly for the generated image and not for the visible field. The curvedmirror 30 can be a double conic for improved image formation. Varioustypes of coatings can be provided on the mirror 30 combiner, including,but not limited to dichroic coatings, metal coatings, such as to providea half-silvered reflector, electrochromatic coatings, anti-reflection(AR) coatings. Mirror 30 can be fully or partially reflective or fullyor partially transparent, with some amount of reflectivity.

Embodiments of the present disclosure provide a measure of distancebetween the image generator (OLED or other spatial light modulatordevice) and the face and temples of the viewer. This helps to preventdiscomfort due to heat where the wearable display is worn for anextended period of time.

The particular arrangement of image-forming components provides suitableimage quality and high resolution to allow reading and other visualactivity involving fine detail.

According to an embodiment of the present disclosure, the optical systemdescribed herein is suitable for applications requiring sensitivity tothe viewer, including not only viewer comfort, but some level of visionmonitoring and adaptation. For example, the apparatus described hereincan be used as part of a system for compensating for vision problems. Byway of example, FIGS. 6C and 7A-7C show various features of anembodiment useful for compensating for macular degeneration. This typeof application can require a measure of viewer monitoring andadaptation, possibly including adjustment of generated data contentsuitable for the viewer.

Gaze Tracking

According to an aspect of the present disclosure, gaze tracking can beprovided as part of the wearable optics system and used to adjust systemparameters according to perceived focus of attention for the viewer.Cameras and infrared (IR) light sources provided on a headset, as shownsubsequently, can provide the gaze-tracking function and correspondingangular measurement data. Gaze tracking can be combined with thecontroller and with a camera image FOV intake. For instance, change ofthe image aspect ratio for generated image data content may beappropriate, allowing the system to adapt image content to thedimensional parameters available from the image generation system. Thus,for example, cameras associated with the HMD can oversample thereal-world input from the object scene, acquiring a wider FOV than canbe displayed by system optics. Gaze tracking identifies the actual FOVavailable to the viewer. The resulting buffered images are related tothe reduced FOV video that can be generated, as controlled by using thesector of the FOV identified using eye gaze recognition.

FIG. 6C is a schematic view that shows an alternate embodiment of thepresent disclosure that employs the imaging path itself for eye trackingand provides 1:1 imaging of the viewer's iris. IR light, or othersensing light, is directed along the optical path, such as generatedfrom, through, or at some other point along the output path of, imagegenerator display 10. Folding surface 26 can be formed as a dichroicsurface, treated to direct the sensing light to beam splitter 24 and tocurved mirror 30 and to the iris of eye E. Sensor light returning fromthe iris generally retraces the light path to prism 20. A portion ofthis returned light from the viewer's eye, as shown by a dashed line inFIG. 6C, can be transmitted, rather than reflected, through surface 26and be conveyed through a complementary facing prism 60 to a trackingsensor 62, which can be a camera, imaging array, or other imaging sensoraccording to an embodiment. This returned light sensed at sensor 62 canbe IR light, for example. The IR light source can be directed to the eyefrom any suitable position, and can be directed through or alongsidecomponents on the optical axis. Surface 26 of the prism is thenconfigured to reflect most of the visible light, performing its turningfunction for image-bearing light, and to transmit IR light suitably forgaze tracking.

Other configurations are possible. Thus, for example, a dichroic coatingcan be employed for surface 26, or some other coating can be employedthat provides the needed redirection by reflection of the bulk ofimage-bearing light, while also allowing sufficient light leakage forsensing.

The embodiment of FIG. 6C can be modified in a number of ways to alloweye gaze tracking through portions of the imaging path. For example,with corresponding changes to components, the use of IR light as asensed light can be replaced by employing visible light. This wouldrequire surface 26 to be partially reflective in the visible range, suchas 90% reflective and 10% transmissive, for example, wherein thetransmitted light is the sensed portion. One or more additional lenses,not shown in FIG. 6C, could be provided in the path of light to sensor62 for gaze tracking using the image path as described herein.

Headset Configurations

FIGS. 7A-7C show a stereoscopic display headset 100 that has separateoptical paths for the left and right eyes of a viewer. As particularlyshown in FIGS. 7A and 7C, the optical relay 40 can be compactly packagedas part of a headset 100 within an optical module 42 that is disposedabove eye level, as the system is normally worn. Beam splitter 24 andcurved mirror 30 are arranged to lie along the visual axis of theviewer. An adjustable strap 102 can be provided to allow adaptation toviewer anatomy. A control logic processor 34 can include the neededelectronics for controlling operation of the optical apparatus;processor 34 components can be mounted above the visual axis anddisposed away from the viewer's forehead. Control logic processor 34 canalternately be separated from, but in wired or wireless signalcommunication with, headset 100.

Eye-tracking can be provided from the headset using the arrangementpreviously described with respect to FIG. 6C or using one or moreeye-tracking cameras 46, working in conjunction with illumination LEDs48, typically infra-red (IR) sources, as shown in FIG. 7B.

FIG. 8 shows a headset 100 that incorporates the optical systemdescribed herein as part of a wearable system for providing userinformation and guidance in performing a task or assignment. Opticalrelay 40 and its associated components are packaged within an opticalmodule 42. A variety of sensors 80, described in detail subsequently,can include one or more SLAM sensors 82, as well as sensors for variousenvironmental or ambient conditions such as temperature, humidity, andthe like, viewer-related, contextual data, cameras, and otherinformation-gathering devices can be integrated with or in wireless orwired signal communication with control logic and related processingcomponents of the HMD headset. Sensors can be provided to support ARimaging, VR imaging, or mixed AR/VR imaging modes of operation.

Connection to power and to signal sources for the headset can beobtained by connection of the headband of headset 100 with externalpower and signal sources, such as other computing and processingequipment worn or carried by the viewer.

System Block Diagram

FIG. 8 is a diagrammatic block diagram illustration showing, for anexemplary embodiment, interrelation of integrated components for an HMDsystem including input sensors 80, output components and systems 90,logic and control components including processors, graphic processingunits (GPU), MVC and other devices. Sensors 80 can includehigh-resolution cameras, multiple displays per eye, 6 to 9 degrees offreedom sensor or other sensors necessary for detection ofhand-gesturing, head-gesturing, voice control, positional location, andestimation or navigation, as well as optical character recognition(OCR), tracking, marker-based or markerless-based AR, location, SLAMsensors, concurrent odometry and mapping sensors, microphones andnoise-cancelling microphones, and any other sensors which could becoupled to and used on an AR/VR headset. Previous figures, for example,gave a diagrammatic illustration of a placement of an IR light for theeye-tracking subsystem. The perspective view of FIG. 9 shows positionsof various sensors, processor, and components coupled to headset 100according to an embodiment of the present disclosure.

Among other sensor technologies which may be housed on the HMD aremanual control inputs. These can include digital buttons, which mayinclude power buttons, and a D-Pad or control-pad for accessing andcontrolling functions by the user, which may or may not be in a dongle;and if not in a dongle then it may exist on the headset or in a wired orwireless remote control. The sensors listed above may include theiroperating systems and output. The control mechanism may also respond toother types of input, including voice command, SLAM, eye tracking, heador hand gesturing, or any other method which can be employed with thesensors and systems mentioned above.

HMD 100 may also house connectors such as power connection forrecharging a battery or for direct connection to an AC source, for theHMD as well as for related input and output devices. There can also beadditional external connectors for HDMI, sound, and other input/outputs,such as additional image overlay display, or for a diagnostics protocolfor upgrading the system. The HMD may also house the microprocessor(s)control circuits. HMD 100 may also contain one or more display per eye,allowing the use of any number of additional projectors, like Picoprojectors, or micro-displays. The displays may be used to projectthough either catoptric system, a dioptric system, or catadioptricsystem, or combinations thereof, such as to generate anultra-short-throw image onto reflective lenses or to project to someother surface, which can be clear plastic, like a polycarbonate resinthermoplastic (Lexan).

The HMD 100 may also house a rechargeable battery which is not typicallyremoved, thus, providing spare energy to continue to power the HMD whenthe removable battery is exhausted or removed. While this battery may besmaller and only have a run-time of several minutes, it can provide theHMD with a “hot-swap” battery system that permits a user to keep viewingfrom the HMD for a time after the removeable battery has died or beenremoved.

HMD 100 may also include a strap and counterweight or other headgear tobalance the HMD and maintain its position on the head. The HMD maycontain a “pinch adjustor” to adjust strap 102. In addition, the HMD mayor may not include a “dongle” whereby one or more of the systems orsubsystems may be connected via wired or wireless to another device,such as could be worn on a belt or carried in a pocket to reduce theoverall weight of the HMD 100. In one embodiment, the HMD may beconnected to another device which is providing power, while in analternative embodiment, the HMD may have its own power from the mains orfrom wireless power transmission or from a battery.

Further, in another embodiment, the HMD may house other subsystems suchas the cameras, the microcontrollers, the connectors, central processingunit, graphics processing unit, software, firmware, microphones,speakers, display, and collector lens; the displays, and othersubsystems.

In another embodiment, the HMD may contain a front facing sensor arrayalong with other sensors mentioned above and optical characterrecognition (OCR) and/or cameras to read and/or measure information fromthe real world object scene. Additionally, the HMD may contain one ormore connectors to connect via wire to the outside world for power anddata (i.e. USB, HDMI, MiniUSB).

Alternatively, some parts of the system mentioned herein may be in adongle attached to the HMD via wire or wireless connection.Alternatively, some portions of the system mentioned herein may becontained in a connected device, like a laptop, smart phone, or Wi-Firouter. Alternatively, some parts of the system mentioned herein may becontained in a smartphone or may be transferred back and forth from asmartphone to the HMD, when synced, such as the HMD displaying thesmartphone apps and other features of the smartphone that wouldotherwise be displayed on the smartphone display. Alternatively, the HMDmay contain and display all the features that a smartphone can.

In another aspect of the present disclosure, HMD 100 may contain all thefeatures of a typical smartphone and no connection may be needed with asmartphone to have all the smartphone features, like web or cellcalling, app use, SMS, MMS, or similar texting, emailing, logging on tothe internet, and the like.

According to an aspect of the present disclosure, the HMD headset mayprovide a computer mediated video shown on the reflective lens layersuch that the wearer may see both the real world and the augmented videoat the same time. In this aspect of the disclosure, such features asvoice/speech recognition, gesture recognition, obstacle avoidance, anaccelerometer, a magnetometer, gyroscope, GPS, spatial mapping (as usedin simultaneous localization and mapping (SLAM)), cellular radiofrequencies, Wi-Fi frequencies, Bluetooth and Bluetooth Lightconnections, infrared cameras, and other light, sound, movement, andtemperature sensors may be employed, as well as infrared lighting, andeye-tracking.

Batteries and other power connections may be needed for various devices,but are omitted from schematic figures for clarity of other features.

SLAM Sensors

Embodiments of the disclosure can further include mechanisms and logicthat provide SLAM (simultaneous localization and mapping) capabilitiesto support the viewer. SLAM uses statistical techniques to map theviewer's environment and to maintain information on the viewer'srelative position within that environment. For example, an image from aSimultaneous Localization and Mapping (SLAM) camera configured for thewearable unit can detect a location of the HMD wearer within the givenenvironment. SLAM capabilities can also be useful where some portion ofthe viewer's FOV is blocked or otherwise obscured. Using SLAM allows thesystem to present portions of the real-world object scene in theelectronically generated image. SLAM capability allows generation anddisplay of image content related to the real-world viewer environment.

To support SLAM capabilities, a headset as shown in the example of FIG.9 can include one or more SLAM sensors 82, such as cameras. In somecases, the SLAM camera may include a visual spectrum camera, an infrared(IR) camera, or a near-IR (NIR) camera. Additionally, or alternatively,the SLAM camera may be include or exclude the viewer from its field ofview. Techniques for providing SLAM capabilities are known to thoseskilled in the mapping arts and can include processing of local imagecontent as well as use of tracking data.

SLAM logic can be provided by control logic processor 34 or by anexternal processor that is in signal communication with processor 34,including processors that are connected to the wearable display deviceby a wired connection or, alternately, processors that are in wirelesscommunication with control processor 34.

IPD Adjustment System

Embodiments of the present disclosure address the need for HMD systemadjustment to viewer anatomy, with benefits for system efficiency andusability. One aspect of variable viewer anatomy relates to inter-pupildistance (IPD). This well-known characteristic relates to overall headdimensions and position of the eye sockets. Mismatch of IPD by thedevice can make it difficult to provide image content and neededfunctions and can make the HMD difficult to wear and use for someviewers. Some systems provide manual methods for IPD adjustment. Asshown in the perspective view of FIG. 10 , the Applicant's HMD canprovide an automated IPD adjustment system 110 that measures the IPD forthe viewer and responds by adjusting the inter-pupil spacing, changingthe distance between the left and right optical modules 421 and 42 r,respectively.

Referring to the FIG. 10 depiction, IPD adjustment system 110 can havean actuator 112 that urges one or both left and right optical modules421, 42 r either farther apart, as shown in distance IPD2, or closertogether as in distance IPD1. This adjustment can be performed atpower-up or other system activation, at operator command, or atpredetermined periods, or at some other time or in response to otherevent. According to an embodiment of the present disclosure, IPDadjustment settings can be stored for each viewer who has previouslyused the HMD. Then, upon entry of information identifying the viewer,the stored IPD adjustment settings can be restored as presets.

There may even be an external device, motorized or mechanical, whicheffects the activation of the preset IPD upon instruction prior towearing the HMD.

In one embodiment, to execute the IPD adjustment function, eye trackingcameras 46 on both left and right optical modules 421, 42 r obtain imagecontent that allows pupil center detection by processor 34 logic. Thislogic determines the relative location of actual pupil centers and IPdistance and determines whether or not the IPD between pupil centers iscompatible with the positioning of left and right optical modules 421,42 r. If positioning is appropriate, no IPD adjustment is necessary.Otherwise, an actuator 112 can be energized to translate one or bothleft and right optical modules 421, 42 r in the horizontal orx-direction as shown in FIG. 10 . At various incremental positions,feedback logic can again measure and calculate any needed adjustmentuntil a suitable IPD is achieved. For IPD adjustment, at least one ofthe left and right optical modules 421 and 42 r can be movable; theother module 421 or 42 r can also be movable or may be stationary.

Adjustable Focus

As a useful default for most virtual image viewability, HMD optics aretypically designed to form the virtual image so that it appears to be atoptical infinity, that is, with rays substantially in parallel to theoptical axis. In an embodiment, the HMD optics are designed to providethe augmented virtual image at about 1 m distance from the viewereyebox, equivalent to approximately 1 diopter prescription glasses.

In some applications, closer focal length is advantageous. To achievethis, the Applicant's solution provides measurement and adjustment fordiopter adjustment of the optical relay 40 optics. Referring to FIG.11A, there is shown a schematic for relay 40 components that includes anactuator 122 and associated components as part of a focal planeadjustment system 120. By changing image generator 10 position along anaxis A, a change in focal position is effected, such as with imagegenerator 10 shifted to the dashed outline position denoted for imagegenerator 10′ in FIG. 11A. This movement causes a corresponding shift ofintermediate image I to the position shown as image I′ in relay 40. Thesplitter 24 and combiner, curved mirror 30, then condition theimage-bearing light to provide a virtual image at a shifted spatiallocation. Actuator 122 can be a linear piezoelectric actuator, forexample, capable of high-speed change between positions. One or moreactuators 122 can be used for moving any of the components describedhereinabove with relation to optical relay 40 in order to adjust theposition of the focal plane.

FIGS. 11B, 11C, and 11D show different views of relay optics with focalplane adjustment system 120. FIG. 11B is a perspective view that showsposition of actuator 122 relative to image generator 10 andcorresponding optics. FIG. 11C is a perspective view from behind imagegenerator 10, showing a piezoelectric actuator 122 mounted to astationary plate 124 behind image generator 10 (not visible in the viewof FIG. 11C). Actuator 122, such as a piezoelectric actuator, translatesimage generator 10 along axis A, which extends orthogonally to theelectroluminescent display component surface of image generator 10. FIG.11D shows a side view of adjustment system 120 components. One or morestabilizing posts 126 has a screw and a compression spring 162 formaintaining image generator 10 in position along the optical path, sothat movement of image generator 10 is constrained to the axialdirection (axis A) during actuation.

One difficulty with the change in focal length relates tovergence-accommodation conflict, as shown in the schematic diagram ofFIG. 12 . Vergence-accommodation conflict (VAC) is a vision phenomenonthat is familiar to developers of three-dimensional (3D) displays andvirtual-reality (VR) display devices. Under real-world normal viewingconditions, where only the object scene is in view as shown in part (a)of FIG. 12 , the viewer's eyes converge, rotating toward one another tofocus on closer objects and, correspondingly, away from one another tofocus on objects at further distance. Accommodation, as the processwhere the lenses of the eyes focus on a close or far away object, isconsistent with convergence for normal viewing in the part (a) depictionand involves epipolar geometry of the HMD cameras and the wearer's eyes.Thus, accommodation and convergence can be considered to be coupled.

Focal plane adjustment system 120 (FIGS. 11A-D) also employs eyetracking sensors and related data for determining when there is adiscrepancy between focal planes for the object world FOV and thegenerated image content. The focal plane position for the generatedimage can be computed according to system optical geometry for thecomponents defined hereinabove. Later adjustments to the geometry,executed by system logic for shifting focal plane position, can berecorded and used to recalculate focal plane position.

In FIG. 12 part (a), the viewer sees a real object, i.e., the viewer'seyes are verged on the real object and gaze lines from the viewer's eyesintersect at the real object. As the real object moves nearer the user,as indicated by the arrow, each eye rotates inward (i.e., converges) tostay verged on the real object. As the real object gets closer, the eye“accommodates” to the closer distance, through reflexive, automaticmuscle movements, by adjusting the eye's lens to reduce the focallength. In this way, accommodation adjustment is achieved. Thus, undernormal viewing conditions in the real world, the vergence distance(d_(v)) equals the accommodation distance (d_(f)).

In 3D displays and VR systems, however, these two processes can bedecoupled, as shown in part (b) of FIG. 12 . FIG. 12 shows an exampleconflict between vergence and accommodation that can occur withstereoscopic three-dimensional displays, in accordance with one or moreembodiments. In this example, an observer is looking at the virtualobject displayed on a 3D electronic display; however, the observer'seyes are verged on and gaze lines mapped from the observer's eyesintersect at the virtual object, which is at a greater distance from theobserver's eyes than the image formed by the 3D electronic display. Asthe virtual object from the 3D electronic display is rendered to appearcloser to the viewer, each eye again rotates inward to remain verged onthe virtual object, but the focus distance of the image is not reduced;hence, the observer's eyes do not accommodate, as in part (a) Thus,instead of increasing the optical power to accommodate for the closervergence depth, the eye maintains accommodation at a display distanceassociated with the 3D electronic display. Thus, the vergence depth (dv)often does not equal the focal length (df) for the human eye for objectsdisplayed from 3D electronic displays. This discrepancy between vergencedepth and focal length is referred to as “vergence-accommodationconflict” or VAC. A user experiencing only vergence or onlyaccommodation, and not both vergence and accommodation, can eventuallyexperience some degree of fatigue, dizziness, discomfort,disorientation, and even nausea in some cases.

In order to compensate and correct VAC, the FIGS. 11A-D relay optics canadjust the relative position of intermediate image I to I′, moving thevirtual image focal plane more closely toward the real-world focalplane.

Model Controller

Embodiments in accordance with the present disclosure may be provided asan apparatus, method, computer program, hardware/software, statemachine, firmware, and/or product. All of the systems and subsystems mayexist, or portions of the systems and subsystems may exist to form theapparatus described in the present disclosure. Accordingly, one or moreportions of the Applicant's solution may take the form of an entirely orpartial hardware embodiment, a predominantly software embodiment(including firmware, resident software, micro-code, etc), or anembodiment combining software and hardware aspects in some combinationthat may all generally be referred to herein, without limitation, as a“unit,” “module,” or “system.” Furthermore, one or more portions orfunctions for the present disclosure may take the form of a computerprogram product or products embodied in any tangible media of expressionor storage having computer-usable program code embodied in or otherwiserepresented using the media. Any combination of one or morecomputer-usable or computer-readable media (or medium) may be utilized,including networked combinations that utilize remote processingcomponents. For example, a random-access memory (RAM) device, aread-only memory (ROM) device, an erasable programmable read-only memory(EPROM or Flash memory) device, a portable compact disc read-only memory(CDROM), an optical storage device, and a magnetic storage device.Computer program code for carrying out operations of the presentdisclosure may be written in any combination of one or more programminglanguages. Further, the intelligence in the main circuitry may besoftware, firmware, or hardware, and can be microcontroller based orincluded in a state machine. The disclosure may be a combination of theabove intelligence and memory and this can exist in a central processingunit or a multiple of chips including a central graphics chip. Thecomputer portion of the disclosure may also include a model viewcontroller (MVC) as shown in FIG. 8 , which is also called herein a“model controller.”

Dithering

According to an embodiment of the present disclosure, dithering can beemployed to modify and improve the visual experience of the viewer.Dithering can be effected, for example, by rapid in-plane vibration of acamera or image generator 10 using a piezoelectric actuator 122, as wasdescribed previously with respect to FIGS. 11A-D. Dithering, imparted tothe displayed image content using synchronous, timed spatialdisplacement, can be a desirable solution for helping to mask oreliminate display-related artifacts, for example.

Dithering can also be used to enhance image resolution using TimedSpatial Displacement. Improved image resolution is goal, and holdspromise for future use of AR/VR glasses in various applications, such asin critical use cases such as surgery visualization. In theseapplications, for example, detail visualization of fine layers and bloodvessels can be critical to successful outcomes. Micro displays continueto mature, with pixel counts of 2 million in a single small device, withfurther improvements likely. These higher resolution displays imposesteeply increased demands on system resources, including higher powerand computation speed and complexity, for example.

The Applicant addresses this problem using dithering of displaycomponents. This solution can make higher resolution images available tousers at a discounted power cost, which in turn can provide lighter,cooler running systems.

Increased pixel resolution is obtained by using the capability to shiftimage generator 10 in-plane, that is, in one or more (x-y plane)directions parallel to the emissive display surface of image generator10, synchronously with corresponding changes in image data content. Withrespect to FIG. 11D, image generator 10 translation is in the x-y plane,orthogonal to axis A (conventionally considered the z-axis) fordithering to increase pixel resolution using synchronous, timed spatialdisplacement. Actuator 122 is configured to provide dithering usingin-plane translation of image generator 10.

As shown schematically in FIG. 13 , multiplication of the imagegenerator 10 resolution is accomplished by physically shifting an array13 of pixels that form image generator display 10 in the x-y plane. Theshift distance is a proper fraction of the pixel pitch. At right in FIG.13 is represented a pixel 130 of array 13, shifted in the x-direction topixel positions 134 and 138 and shifted in the y-direction to positions132 and 136. Synchronous with the shift action is modulation of imagedata for each pixel 130 at its original position and at each shiftedposition 132, 134, 136, and 138. Thus, for example, the state of a pixelat its position 134 (e.g., brightness or color) can differ from itsstate at position 138, according to the image data content that isprovided with the shift. With a half-pixel shift in each x- andy-direction, for example, the effective pixel count can be increased byat least 4 times. With a half-pixel shift only along one axis, such asonly along the x or y axis as shown, the effective resolution along theaxis parallel to the shift can be doubled. Overall, the power cost ofnano scale piezo shifting is much lower than the cost to design andimplement 4 x the pixel count using a higher resolution image generator10 element. An array 14 represents increased pixel resolution.

For the embodiment of FIG. 13 , an image generator 10 was provided,having a 240 frames per second (fps) refresh rate. In terms of thepiezoelectric actuation provided, each pixel element can be relocated at½ the delta of the pixel element center-to-center distance in the array13. This arrangement can provide 60 fps display at 4× the resolution ofthe original image generator 10.

By way of example, an embodiment of the present disclosure employsQNP-XY Series Two-Axis, CY Piezo Nanopositioners for image generator 10dithering actuation.

A number of features allow piezoelectric dithering to provide enhancedresolution, including the following: (1) Adjustable travel range, suchas from 100 um to 600 um, for example; (2) Long device lifetimes; (3)Superior positioning resolution; and (4) High stiffness and otherfactors.

A number of piezoelectric actuators provide the option of closed-loopfeedback that allows sub-nanometer resolution and high linearity.

Dithering for increased resolution can utilize any of a number ofmovement patterns for in-plane displacement of pixels. Patterns fordithering the pixels in the display (or the displays) for increasedresolution include transposing with rectilinear motion, curvilinearmotion, or in a translational, rotational, periodic, or non-periodicmotion, or any combination of the above. Also, the pattern could includea rectangle pattern which may increase the resolution by 4 times.Another alternative way to address the pixel movement is to have pixelsthat are approximately the same size as the non-emissive dark or “black”space between the pixels where the dithering translation of each pixelin a display is dithered to the next adjacent unused space existingbetween the pixels in the display.

For viewer comfort, a strap adjustment can be provided, allowing both aone-time fastener positioning adjustment and a flexible stretch band.

The invention has been described in detail with particular reference toa presently preferred embodiment, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the disclosure. The presently disclosed embodiments are thereforeconsidered in all respects to be illustrative and not restrictive. Thescope of the invention is indicated by any appended claims, and allchanges that come within the meaning and range of equivalents thereofare intended to be embraced therein.

What is claimed is:
 1. A wearable display apparatus comprising: (a) animage generator that forms a 2D image; (b) a partially transmissivemirror having a curved reflective surface; (c) a beam splitter disposedto reflect light toward the curved mirror surface; and (d) an opticalimage relay that is configured to conjugate the formed 2D image at theimage generator to a curved focal surface of the partially transmissivemirror, wherein the curved focal surface is defined between the curvedreflective surface of the partially transmissive mirror and the beamsplitter, wherein the optical image relay comprises: (i) a prism havingan input surface, an output surface, and a folding surface extendingbetween the input and output surfaces and configured for folding anoptical path for light generated by the image generator, wherein anaperture stop for the relay lies within the prism; (ii) at least a firstplano-aspheric lens in optical contact against the prism input surfaceand configured to guide light from the image generator toward thefolding surface; wherein the relay, curved mirror, and beam splitter areconfigured to form an exit pupil for viewing the 2D image superimposedon a portion of a visible object scene.
 2. The apparatus of claim 1wherein the wearable display apparatus is configured to form abinocular, 3-dimensional image.
 3. The apparatus of claim 1 wherein theoptical image relay further comprises a second plano-aspheric lens inoptical contact against the prism output surface and configured todirect the light to the curved focal surface.
 4. The apparatus of claim3 further comprising a singlet and a doublet for shaping the beam ofimage-bearing light directed toward the prism.
 5. The apparatus of claim1 further comprising a field lens that directs an image-bearing lightbeam from the image generator toward the first plano-aspheric lens. 6.The apparatus of claim 1 having a field of view (FOV) wherein a ratio ofhorizontal:vertical FOV dimensions equals or exceeds 1.5 and wherein thehorizontal FOV equals or exceeds 50 degrees, and wherein the opticalpath is folded within the prism so that a portion of the system opticalaxis between the image generator and the prism is parallel to a linebetween left and right pupils of a viewer.
 7. The apparatus of claim 1wherein the curved surface is conic.
 8. The apparatus of claim 1 whereinthe curved surface is spherical.
 9. The apparatus of claim 1 wherein thebeam splitter is a polarization beam splitter.
 10. The apparatus ofclaim 9 further comprising a quarter wave plate in the path of lightbetween the polarization beam splitter and the curved mirror.
 11. Theapparatus of claim 1 wherein the beam splitter is a wire gridpolarization beam splitter.
 12. The apparatus of claim 11 wherein thequarter wave plate is cylindrically curved.
 13. The apparatus of claim 1wherein the image generator is an electroluminescent display that isdisposed to direct an image-bearing light beam in a direction parallelto a line between left and right pupils of a viewer.
 14. The apparatusof claim 1 wherein the folding surface is at least 10% transmissive andfurther comprising an eye-tracking sensor in the path of light thatreflects from an iris of the viewer and transmits through the foldingsurface.
 15. A wearable display apparatus having a left-eye opticalsystem and a right-eye optical system, wherein each optical systemdefines a corresponding exit pupil for a viewer along a view axis andcomprises: (a) an electroluminescent image generator that is energizableto direct image-bearing light for a 2D image from an emissive surface;(b) a curved reflective surface disposed along the view axis andpartially transmissive, wherein the curved reflective surface defines acurved intermediate focal surface; (c) a beam splitter disposed alongthe view axis and oriented to reflect light toward the curved reflectivesurface; and (d) an optical image relay that is configured to opticallyconjugate the formed 2D image at the image generator with theintermediate focal surface, wherein the optical image relay comprises:(i) a prism having an input surface facing toward the emissive surfaceof the image generator, an output surface facing toward the intermediatefocal plane, and a folding surface extending between the input andoutput surfaces and configured for folding an optical path for lightgenerated by the image generator, wherein an aperture stop for the relaylies within the prism; (ii) a first plano-aspheric lens in opticalcontact against the prism input surface and configured to guide theimage-bearing light from the image generator toward the folding surface;(iii) a second plano-aspheric lens in optical contact against the prismoutput surface and configured to direct the light from the foldingsurface toward the curved intermediate focal surface; wherein the relay,curved mirror, and beam splitter are configured to form the exit pupilfor viewing the generated 2D image superimposed on a portion of avisible object scene.
 16. The apparatus of claim 15 further comprising acylindrically curved quarter wave plate in the path of light between thebeam splitter and the curved reflective surface.
 17. The apparatus ofclaim 15 wherein the folding surface is partially transmissive andfurther comprising an eye-tracking sensor in the path of light thatreflects from an iris of the viewer and transmits through the foldingsurface.
 18. A method for forming an image for a viewer of a wearabledisplay apparatus, the method comprising: a) forming an eyebox fordisplay, from an intermediate image formed on a curved aerial surface,by disposing a beam splitter to reflect light from the curved aerialsurface toward a partially transmissive curved surface, wherein thecurved aerial surface is a focal surface for the partially transmissivecurved surface; b) configuring an optical image relay to conjugate thecurved aerial surface with a two-dimensional image generator, whereinthe optical image relay is configured to: (i) direct modulated lightfrom the image generator along an optical path; and (ii) focus the lightthrough an aperture stop that lies within a turning prism, wherein theturning prism has an input surface, an output surface, and a foldingsurface extending between the input and output surfaces and isconfigured for folding the optical path to direct the modulated light tofocus along the curved aerial surface; and c) providing one or moresignals to the image generator for providing modulated light output,wherein the relay, curved mirror, and beam splitter are configured toform an exit pupil within the eyebox for viewing the two-dimensionalimage superimposed on a portion of an object scene that is visiblethrough the partially transmissive curved surface.
 19. The method ofclaim 18 wherein configuring the optical image relay comprises providingat least one aspheric corrector coupled to the input surface of theprism.
 20. The method of claim 18 further comprising obtaining an imageof the viewer's iris from light transmitted through the folding surface.21. The method of claim 18 further comprising adjusting a focal surfaceposition for the modulated light using a piezoelectric actuator.